Vision-based Detection of Acoustic Timed Events: a Case Study on Clarinet Note Onsets

Acoustic events often have a visual counterpart. Knowledge of visual information can aid the understanding of complex auditory scenes, even when only a stereo mixdown is available in the audio domain, \eg identifying which musicians are playing in large musical ensembles. In this paper, we consider a vision-based approach to note onset detection. As a case study we focus on challenging, real-world clarinetist videos and carry out preliminary experiments on a 3D convolutional neural network based on multiple streams and purposely avoiding temporal pooling. We release an audiovisual dataset with 4.5 hours of clarinetist videos together with cleaned annotations which include about 36,000 onsets and the coordinates for a number of salient points and regions of interest. By performing several training trials on our dataset, we learned that the problem is challenging. We found that the CNN model is highly sensitive to the optimization algorithm and hyper-parameters, and that treating the problem as binary classification may prevent the joint optimization of precision and recall. To encourage further research, we publicly share our dataset, annotations and all models and detail which issues we came across during our preliminary experiments.Comment: Proceedings of the First International Conference on Deep Learning and Music, Anchorage, US, May, 2017 (arXiv:1706.08675v1 [cs.NE]

Music Information Retrieval beyond Audio: A Vision-based Approach for Real-world Data

Digital music platforms have recently become the primary revenue stream for recorded music, making record labels and content owners increasingly interested in developing new digital features for their users.Besides listening to expert-curated playlists and automatically recommended music, users can also benefit from a more informative, non-linearly accessible experience accommodating multiple perspectives on the content.To give some examples of such enriched experiences, an alternative version of a piece can automatically be suggested. Users can skip throughout a long classical music piece guided by a visualization of its structure (\eg movements, recurring themes). They can also switch viewpoints while watching a music video instead of sticking to the editor's choice.Developing such features requires innovation of automated content-based methods that extract musical knowledge. Traditionally, Music Information Retrieval (Music IR) researchers have tackled this problem mostly from an audio-only perspective.Several works have however shown that other types of data, such as social tags, listening behaviors, and symbolic music scores, can largely improve the performance of audio-only algorithms, or even enable tasks that cannot be solved at all using audio alone.In this thesis, we focus on the relatively unexplored field of \textit{vision-based Music IR}, which studies how to analyze the visual channel accompanying a music recording in order to learn more about the music piece being performed.Several existing methods require obtrusive settings, such as 3D motion capture systems, which are not applicable in professional environments (\eg during a live classical music concert). Other methods rely instead on favorable viewpoints, static cameras, and uniform backgrounds to simplify the musicians' movements analysis process.In both cases, the devised algorithms may not be suitable for commercial music platforms, especially those dealing with \textit{real-world data} --- \ie \textit{unstructured} and \textit{unconstrained} music videos.We therefore consider tasks, algorithms and datasets with the real-world data challenges in mind, advancing the state-of-the-art in two ways: (i) we investigate how to process videos of a single musician aiming to extract musically relevant cues that can be exploited to solve existing, as well as new, Music IR problems, and (ii) we address the challenging case of large ensembles, proposing a way to possibly parse complex scenes and link musician-wise cues to identity and instrumental part annotations.More in detail, this thesis first presents a global motion feature which aims to represent musicians' movements over time.While lightweight and instrument-generic, it shows limitations with camera motion.For this reason, we switch to detecting ``play\-ing/non-playing'' (P/NP) labels, which can be guessed from different viewpoints and at different scales and they can be used to encode the instrumentation of a performance over time.We first show the value of such semantic feature by proving that it allows to roughly synchronize a symbolic music score to a performance recording.We then focus on the visual analysis of large classical music ensembles videos, presenting a semi-automatic framework for P/NP annotation.The experiments show that video face clustering is a critical problem to solve; we therefore illustrate a novel method that exploits the \textit{quasi-static scene} properties of classical music videos to generate better face clusters by relying on an automatically built map of the scene.Finally, we address the challenging problem of detecting note onsets for clarinetist videos as a case study for woodwind and brass instruments. We propose a novel convolutional network architecture based on multiple streams and absence of temporal pooling, aiming to capture the fine spatio-temporal information conveyed by finger movements.Our proposed methods, outcomes, and envisioned applications show that real-world music videos are an unexploited asset rather than a problem to avoid.Furthermore, the light this thesis sheds on vision-based Music IR gives various indications on where future Computer Vision and Music IR research agendas can meet, bringing further innovation to the digital music platforms market.Multimedia Computin

On detecting the playing/non-playing activity of musicians in symphonic music videos

Information on whether a musician in a large symphonic orchestra plays her instrument at a given time stamp or not is valuable for a wide variety of applications aiming at mimicking and enriching the classical music concert experience on modern multimedia platforms. In this work, we propose a novel method for generating playing/non-playing labels per musician over time by efficiently and effectively combining an automatic analysis of the video recording of a symphonic concert and human annotation. In this way, we address the inherent deficiencies of traditional audio-only approaches in the case of large ensembles, as well as those of standard human action recognition methods based on visual models. The potential of our approach is demonstrated on two representative concert videos (about 7 hours of content) using a synchronized symbolic music score as ground truth. In order to identify the open challenges and the limitations of the proposed method, we carry out a detailed investigation of how different modules of the system affect the overall performance.harvest http://mmc.tudelft.nl/content/detecting-playingnon-playing-activity-musicians-symphonic-music-videosMultimedia Computin

A pharmacokinetic/clinical approach to postulate a local action of intra-articular xylazine administration in the horse: a preliminary study

The study aims to evaluate whether the analgesic effect of intra-articular (IA) route of xylazine administered to horses following arthroscopic surgery is due to a local or a systemic action. Two connected studies were performed. In the first, 1mg/kg b.w. of xylazine was injected IA, and blood samples were taken to assess drug systemic absorption. In addition, systemic effects of the drug (sedation, ataxia or reduction of respiratory and cardiac rate) were registered. Control horses injected with saline IA were included in the study to exclude the influence of anaesthesia in the occurrence of these manifestations. In the second study, 1mg/kg b.w. of xylazine was administered intravenously (i.v.) in healthy horses. Blood samples were collected to determine the concentrations of xylazine, and the same signs of systemic effects of the drug were recorded. By correlating these parameters, a systemic no effect' concentration was defined. Pharmacokinetic data after IA administration resulted in some xylazine absorption (bioavailability equal to 58.12%) with values above the systemic no effect' concentration. The occurrence of some signs related to systemic effects in horses receiving IA xylazine was significant compared with horses receiving saline. In conclusion, a systemic action of the drug after IA administration cannot be excluded

Successful antiviral therapy determines a significant decrease in squamous cell carcinoma antigen-associated (SCCA) variants' serum levels in anti-HCV positive cirrhotic patients.

Aberrant squamous cell carcinoma antigen (SCCA) expression is an early event in hepatocarcinogenesis, and increasing serum levels of SCCA variants IgM immune complexes (SCCA-IgM IC) have been found in cirrhotic patients developing hepatocellular carcinoma (HCC). We longitudinally evaluated a cohort of cirrhotic patients with hepatitis C virus infection (HCV) who underwent pegylated interferon (PEG-IFN) and ribavirin treatment. SCCA-IgM IC levels were assessed in the sera of 33 cirrhotic patients with HCV (21 males, median age 57 years) before, at the end and at 6-month and 1-year follow-up after treatment with PEG-IFN and ribavirin. SCCA-IgM IC serum levels (arbitrary units/mL, AU/mL) were evaluated according to treatment outcome: sustained virological response (SVR) vs nonresponse (NR). Overall, 15 patients obtained a SVR to antiviral therapy (45%). There was no significant difference in baseline SCCA-IgM IC serum levels between SVR and NR patients. When compared to baseline (451.2 AU/mL), SVR patients showed a significant decrease in median SCCA-IgM IC serum levels at the end of treatment (186.8 AU/mL, P = 0.013) and at both 6-month (96.8 AU/mL, P < 0.001) and 1-year follow-up (52.4 AU/mL, P < 0.001), while no significant modification was observed in NR patients. In patients with HCV-related liver cirrhosis, successful antiviral therapy is associated with a dramatic and significant decrease in SCCA-IC serum levels. Because of the pathophysiological correlation between SCCA and liver carcinogenesis, it is hypothesized that in patients with liver cirrhosis, SVR may be accompanied by a decreased proliferative stimulation

Small intestinal bacterial overgrowth in patients suffering from scleroderma: clinical effectiveness of its eradication.

OBJECTIVES: After the skin, the gastrointestinal tract is the second most common target of systemic sclerosis (SSc). AIM: Our aims were to investigate orocecal transit time (OCTT) and the presence of small intestinal bacterial overgrowth (SIBO) in SSc as a cause of intestinal symptoms. METHODS: Fifty-five SSc patients and 60 healthy controls, sex and age matched, entered the study. Enrolled subjects completed a questionnaire for intestinal symptoms and a global symptomatic score (GSS) was calculated. OCTT and the presence of SIBO were assessed by a lactulose breath test (LBT). Patients with SIBO were treated with rifaximin 1,200 mg/day for 10 days. Finally, a second questionnaire and LBT were performed 1 month after the end of therapy. RESULTS: The prevalence of SIBO was higher in SSc patients compared with controls (30/54 vs 4/60, respectively, P < 0.001). OCTT was significantly slower in SSc patients compared with controls (150 min, 25-75th percentile 142.5-165 vs 105 min, 25-75th percentile 90-135, respectively, P < 0.001). In patients with SIBO, the median GSS score was 8 (25-75th percentile 3.25-10.75). Eradication of SIBO was achieved in 73.3% of patients, with a significant reduction of symptoms in 72.7% of them (GSS score 2, 25-75th percentile 1-3, P < 0.05). CONCLUSIONS: These data suggest that SIBO occurs more frequently in SSc patients than in controls. Intestinal symptoms in these patients may be related to this syndrome and its eradication seems useful to improve clinical features. OCTT is significantly delayed in SSc patients, suggesting an impairment of intestinal motility, a further risk factor for the development of SIBO

Small intestinal bacterial overgrowth in patients suffering from scleroderma: clinical effectiveness of its eradication

OBJECTIVES: After the skin, the gastrointestinal tract is the second most common target of systemic sclerosis (SSc). AIM: Our aims were to investigate orocecal transit time (OCTT) and the presence of small intestinal bacterial overgrowth (SIBO) in SSc as a cause of intestinal symptoms. METHODS: Fifty-five SSc patients and 60 healthy controls, sex and age matched, entered the study. Enrolled subjects completed a questionnaire for intestinal symptoms and a global symptomatic score (GSS) was calculated. OCTT and the presence of SIBO were assessed by a lactulose breath test (LBT). Patients with SIBO were treated with rifaximin 1,200 mg/day for 10 days. Finally, a second questionnaire and LBT were performed 1 month after the end of therapy. RESULTS: The prevalence of SIBO was higher in SSc patients compared with controls (30/54 vs 4/60, respectively, P &lt; 0.001). OCTT was significantly slower in SSc patients compared with controls (150 min, 25\u201375th percentile 142.5\u2013165 vs 105 min, 25\u201375th percentile 90\u2013135, respectively, P &lt; 0.001). In patients with SIBO, the median GSS score was 8 (25\u201375th percentile 3.25\u201310.75). Eradication of SIBO was achieved in 73.3% of patients, with a significant reduction of symptoms in 72.7% of them (GSS score 2, 25\u201375th percentile 1\u20133, P &lt; 0.05). CONCLUSIONS: These data suggest that SIBO occurs more frequently in SSc patients than in controls. Intestinal symptoms in these patients may be related to this syndrome and its eradication seems useful to improve clinical features. OCTT is significantly delayed in SSc patients, suggesting an impairment of intestinal motility, a further risk factor for the development of SIBO